Methods for extracting objects from digital images and for performing color change on the object

ABSTRACT

Operating a system displaying a mirror-mimicking image on the monitor to enable a user to try a clothing item, by: obtaining digital images of a live video feed of length of N seconds from the camera; saving the digital images in the memory; flipping the images about a vertical axis so as to reverse right and left sides of the image; applying a transformation mapping to the images to modify the images such that they appear to mimic a reflection of a mirror, to thereby obtain transformed images; displaying the transformed images on the monitor to thereby replay the video feed of N seconds using the transformed images; wherein the length of N seconds is configured to be enough time to allow the user to get a good impression on the way the clothing item looks.

CROSS-REFERENCE(S) TO RELATED APPLICATIONS

This application is a Continuation of and claims the benefit of, U.S.patent application Ser. No. 14/253,819, filed Apr. 15, 2014, which is aContinuation-in-Part of U.S. application Ser. No. 13/843,001, filed Mar.15, 2013 (now U.S. Pat. No. 8,982,109 issued Mar. 17, 2015), which is aContinuation-in-Part of U.S. application Ser. No. 13/088,369, filed Apr.17, 2011 (now U.S. Pat. No. 8,624,883 issued Jan. 7, 2014), which is acontinuation of U.S. patent application Ser. No. 11/817,411, filed Aug.30, 2007 (now U.S. Pat. No. 7,948,481 issued May 24, 2011), which is aNational Phase Application of PCT International Application No.PCT/IL2006/000281, International Filing Date Mar. 1, 2006, which claimsthe benefit of U.S. Provisional Application No. 60/656,884, filed Mar.1, 2005 and U.S. Provisional Application No. 60/656,885, filed Mar. 1,2005. U.S. patent application Ser. No. 13/843,001 further claims thepriority benefit of U.S. Provisional Application No. 61/738,957, filedDec. 18, 2012. U.S. patent application Ser. No. 14/253,819 furtherclaims the priority benefit of U.S. Provisional Application No.61/892,368, filed Oct. 17, 2013 and U.S. Provisional Application No.61/862,025, filed Aug. 4, 2013. The entire disclosures of all of theabove listed applications are incorporated herein by reference.

TECHNICAL FIELD

The invention relates generally to imaging and display systems and, moreparticularly, to monitors, and interactive displays, e.g., in retailand/or service environments, medical or home situations, videoconferencing, gaming, and the like. Specific implementations relate tovirtualizing a mirror in situations where users expect to see a mirror,e.g., in trying on apparel. Another specific implementation relate tosituations where a natural appearance is preferable over standard videoimage, such as in, e.g., video conferencing.

BACKGROUND

The conventional mirror (i.e., reflective surface) is the common andmost reliable tool for an individual to explore actual self-appearance,in real time. A few alternatives have been proposed around thecombination of a camera and a screen to replace the conventional mirror.However, these techniques are not convincing and are not yet accepted asa reliable image of the individual as if he was looking at himself in aconventional mirror. This is mainly because the image generated by acamera is very different from an image generated by a mirror.

In U.S. application Ser. No. 13/843,001, we proposed a novel technology(computerized method) to address the challenges of converting andtransforming a still image or 2D/3D video created by one or morecameras, with or without other sensors, into a mirror or videoconference experience.

SUMMARY

The following summary of the disclosure is included in order to providea basic understanding of some aspects and features of the invention.This summary is not an extensive overview of the invention and as suchit is not intended to particularly identify key or critical elements ofthe invention or to delineate the scope of the invention. Its solepurpose is to present some concepts of the invention in a simplifiedform as a prelude to the more detailed description that is presentedbelow.

In this disclosure we describe a computerized technique that takes avideo, a still, or a group of still images, before or after thetransformation described in U.S. application Ser. No. 13/843,001, andcontinue the computerized method to address additional functionalities,such as user interface, augmented reality, color change, texture change,shape manipulation of an object, body or background, and more. Inaddition, the calibrated virtual camera approach allows for body line,body curve and body type measurements from a 2D or 3D image or video.

For the augmented reality capabilities, first a 2D and/or 3D model ormask will be created for one or more items or elements (e.g., shirt,dress, pants, head, shoe, glass, entire body, and the like), then themodel or mask manipulation can enable the augmented reality featureslike color, texture replacement and manipulation, or geometricmeasurement, or complete object replacement. The disclosed technique isdifferentiated from other techniques that are using only 3D camera (IRor dual camera). In the following disclosed techniques one can build themodel from a single regular camera and can improve the results withmultiple cameras and additional information from IR, 3D camera or othersensors and information regarding the object sought to be manipulated.

Some embodiments can include multi-level user identification.Specifically, the embodiments can include face recognition improvement,user learning, and adaptation. Known methods of face recognition canutilize algorithms that can be very sensitive to face pose. In thepresent technique, a smooth experience of face pose can be created toaccommodate different camera locations and angles.

Various embodiments can include code identification, e.g., quickrecognition (QR) code, 1D Code, hidden code and the like. Theembodiments can be adapted to discover codes from a relatively longdistance with a relatively small image as compared to the image underprojection or under other optic distortion. Also, it may includewireless identification, e.g., NFC, WiFi wideband, microwave 3D, bodyaccess network (BAN) chip and the like. Wireless identification can bemade from a mobile device, watch glass, microchip or any other carry ondevice or microchip. Other body measurement techniques may includefingerprinting, body identification, body type, eyes, palm recognition,X-ray correlation, body temperature, body pulse, blood pressure and thelike.

In another aspect a user interface and methods to control and operatemanually or automatically the virtual mirror capabilities are described.

In additional embodiments the product mechanical design and appearanceare disclosed, to enhance the usability and functionality and the userexperience overall.

In some embodiments, a non-transitory computer-readable storage mediumfor operating a monitor, a camera, and a processor, is provided andconfigured so as to display a mirror-mimicking image on the monitor, andcomprising: on a device having the processor and a memory, storing aprogram for execution by the processor, the program includinginstructions for: capturing an image or video from the system;calibrating the image or video based on a user tracking system;detecting an edge of an object present in the captured image or video;measuring a characteristic of the object; and creating a measurementsignature of the object. In some embodiments, computer implementedmethod for operating a system having a monitor, a camera, and aprocessor is provided and configured so as to display a user's image onthe monitor, and comprising: on a device having the processor and amemory storing a program for execution by the processor, the programincluding instructions for: capturing an image or video from the system;identifying an element from the image or video; selecting the identifiedelement; rendering an original model of the selected element; selectinga new appearance for the selected element; and rendering a new modelbased on the original model and the selected new appearance for theelement. The processor performs distortion transformation that comprisesoperation to generate a user experience that simulates the user lookinginto a traditional reflective mirror. The distortion transformation mayinclude the steps: obtaining a digital image from the camera; flippingthe image about a vertical axis so as to reverse right and left sides ofthe image; applying a transformation mapping to the image to modify theimage such that it appears to mimic a reflection of a mirror; resizingthe image to reduce variations caused by changes in the object'sdistance to the camera; and displaying the image on the monitor afterperforming the flipping, transformation mapping, and resizing.

In another aspect, a computerized method includes instructions for atleast one from the group consisting of: complex stitching based on auser's point of view and a location in front of the monitor; image orvideo analyzing to better utilize an inventory of a store and to supportan ecommerce platform; correcting optical distortion of the image orvideo; utilization of multiple cameras to provide additionalinformation; and presentation of the model locally or at a remotelocation on a mobile device or on a stationary monitor.

In a further aspect, a method for obtaining measurements of selectedobject in the image comprises at least one from the group consisting of:measuring in pixels and transforming the measured pixels to a known unitof measurement based on a perimeter model, multiple 2D cuts, a 3D model,statistical optimization, comparison with a previously obtained set ofmeasurements pertaining to the user, determination of a point of focusfor the measuring based on one or more characteristics of the user,determination of a point of focus for the measuring based on a height ofthe user, determination of a point of focus for the measuring based on agender of the user, determination of a zone corresponding to the likelybust/chest of the user based on a height and gender of the user, anddetermination of a zone corresponding to the likely navel of the userbased on a height and gender of the user.

In other aspects, a method for creating of a measurement signature of anobject in the image comprises at least one from the group consisting of:storing analyzed data as a vector with weighting allowing for single ormultiple attribute scoring comparisons between users, recording aprocessed image for future comparison, recording a processed image forfuture deduction of additional information, recording a processed imageto allow for future image correlation techniques, addition of pointersto the image to facilitate correlation, and addition of lines to theimage to facilitate correlation.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated into thisspecification, illustrate one or more exemplary embodiments of theinventions disclosed herein and, together with the detailed description,serve to explain the principles and exemplary implementations of theseinventions. One of skill in the art will understand that the drawingsare illustrative only, and that what is depicted therein may be adaptedbased on the text of the specification and the spirit and scope of theteachings herein.

In the drawings, where like reference numerals refer to like referencein the specification:

FIG. 1 illustrates an embodiment for augmented reality to simulate amirror appearance on a flat screen display;

FIG. 2 illustrates an embodiment of an augmented reality module;

FIG. 3 illustrates an embodiment of an augmented reality module that canreplace a body part, color, orientation and texture of an item or objectin the foreground or background of the image;

FIG. 4 illustrates an embodiment for computing methods to create a modelfor color and texture exchange;

FIG. 5 illustrates an example of a method to extract a body measurementfrom an image, multiple images or a short video;

FIG. 6 illustrates an example of a security system application formultilevel user learning and authentication;

FIG. 7 illustrates an example of a security system application forparallel or in-band camera manipulation;

FIGS. 8a-8c illustrates schematics for sharing mirror experience in avideo/voice conference;

FIG. 9 illustrates an example for a 2D input to the model generator;

FIG. 10 is an example of a 2D model of the upper shirt from the image ofFIG. 9;

FIG. 11 is a schematic illustrating classification of male body types;

FIG. 12 is a schematic illustrating classification of female body types;

FIG. 13 depicts an example of the result of additional processes thateliminate the hand from the body and point to different intervals ofwhere to look for particular body measurements;

FIG. 14 depicts an example of transformation on a camera stream tocreate a mirror effect on the background environment;

FIG. 15 illustrates an example of a virtual mirror with user recognitionand authentication and with user interface;

FIG. 16 illustrates an example of a virtual mirror with split screenmode and thumbnails; and

FIG. 17 illustrates an embodiment of user flow of usage.

DETAILED DESCRIPTION

The following examples illustrate some embodiments and aspects of theinvention. It will be apparent to those skilled in the relevant art thatvarious modifications, additions, substitutions, and the like can beperformed without altering the spirit or scope of the invention, andsuch modifications and variations are encompassed within the scope ofthe invention as defined in the claims which follow. The followingexamples do not in any way limit the invention. For ease ofcomprehending the various embodiments, the description includesdescription of various features under specific headings. However, thesefeatures may be implemented individually or in a mix-and-match fashionso as to take advantage of specific feature(s) for specific application.

General Overview

Embodiments of the present invention utilize a camera and a flat paneldisplay to provide the user with the experience of looking at a mirror.That is, the image stream from the camera is manipulated and transformedsuch that when it is displayed on the screen, it appears to mimic areflection in a mirror, rather than a recorded video stream. Since the“virtual mirror” image is digitally produced, it can be stored andmanipulated to provide further enhancement features. For example, theimage can be manipulated to change colors, to enable multi-level useridentification and authentication, and to enable body measurements. Thesystem includes a calibrated camera, such that the system is capable ofproviding augmented reality features, for example, color, texture, shapemanipulation of body items or background in, e.g., a virtual mirror orvideo conference applications. It is also capable of performing bodymeasurements and body monitoring for commercial, security, andhealthcare applications.

Regarding the hardware, various embodiments may include single ormultiple cameras from all types of sensors, including but not limitedto, CCD, CMOS, IR CMOS and the like, and format protocols includingHDMI, Firewire, USB, IP, HDbaseT wireless and the like, and anyavailable resolutions including HD, regular SIF, 4k, 8k and the like.The display can include single or multiple screens or virtual screen,and regular screens in all sizes, shapes and proportions. The displaycan include mirror like screens, smartphones or tablets, projectors,holograms, 3D screens, Google glass, projector head mount display (HMD),glass (with switchable obscure layer) and the like.

In the disclosed embodiments the camera can be located anywhere. A bestpractice is to provide the camera above the screen facing the user.Additional locations can include the bottom of the screen, the sides ofthe screen or behind the screen if the screen is a bidirectional screen.In the case of 3D, two cameras can, for example, be provided at the topand side by side, which can provide superior results and ease thecomputerized method. Also, 3D can be manipulated from a single cameraonce the user is moving and/or rotating.

The embodiments can include a computerized method to cover a full bodyor a partial body, and different scaling per user selection (e.g.,correction of head and/or eyes, direction of view/point of view and thelike).

Each of the disclosed embodiments can be provided over a 2D or 3D stillhard copy image, a 2D or 3D still digital image, over analog/digitalvideo that was recorded by analog/digital camera or by frame grabberHDMI (or equivalent), IP, USB, Firewire (wired or wireless link) and thelike. Each of the disclosed embodiments can be provided over digitalstreaming video that can be delivered to a computing device over anysuitable communication method such as USB, HDMI, IP cat-5/fiber, WLAN,Firewire, HDbaseT, any combination of the above on a single cable or onmultiple cables. Each of the disclosed embodiments can be provided whenthe computing device resides in the camera, in the screen, in the cloud,or in a computer including a workstation, server and the like.

Each of the disclosed embodiments can be provided when the computingdevice is distributed between the system elements, e.g., thecomputerized method can reside partially in the camera, partially in thevideo acquisition element, partially in the screen graphic processingunit (GPU), partially in the cloud, partially in the user smartphonedevice or any suitable combination thereof. Each of the disclosedembodiments can be provided when the computing device resides in asmartphone, tablet, notebook, screen, camera, HMD, Google glass and thelike.

Each of the disclosed embodiments can be provided when the video ispreprocessed to extract a 3D model of a selected item, where thepreselecting can be done manually by the user or automatically byapplying rules for selection. The model extraction can be performedlocally by DSP/CPU next to the camera, in the cloud or in the userapplication on the user's device. It is more efficient to extract themodel in a GPU (graphics processing unit) for a single object or a stillimage; in this case, a regular CPU can be sufficient.

Each of the disclosed embodiments can be provided when the rendering isbeing performed locally in the augmented reality station, in the cloud,or at the user mobile device. The rendering may be performed in the GPUfor a single image or a very short video; in these cases, a CPU canprovide sufficient performance.

Each of the disclosed embodiments can be provided when enhancedrendering techniques are performed to improve video quality, e.g.,interpolating the object and the image to higher resolution anddecimating after combining to smooth the edges and eliminate an aliasingeffect. The present invention can eliminate flickering edges, e.g., bysmoothing the model frame by frame, by eliminating the allowed changesper frame on the edges, by smoothing edges per frame, by applyingaveraging filters on single or multiple frames, and the like. Additionalimprovements can be achieved by increasing the frame rate and applyingadditional smoothing technique on the added frame to smooth the effectthat would occur on the original frame rate.

Each of the disclosed embodiments can be provided regardless of whetherscreen resides near the camera. For example, the screen can be locatedin a control room. A camera main stream or recorded information can beaccessed via remote control, via other address directly with physicalinfrastructure, or over the cloud.

In one embodiment the control on the station functionality can bethrough a gesture, eye movement, palm movement, finger movement or braincontrol. The present invention can be controlled with additionalaccessories like a laser beam pointer, voice control, wireless, orultrasound tracing.

Segmentation

Accurate segmentation of objects and/or elements out of an image orvideo is one of the biggest challenges in the present field. Knowntechniques for solving this problem involved a clear tradeoff betweenaccuracy and speed. Prior to the present invention, there is nosatisfactory solution to the problem. Use of an infrared (IR) camerawith three-dimensional (3D) depth measurement is one approach in theindustry; however, quality of the approach suffers. For example, edges,image distortion, and/or low resolution misalignment between the IR andregular cameras prevent successful and convincing segmentation.Chromatic separation and edge detection techniques are very demandingfrom a computer resource perspective and do not yield a consistentperformance. Some techniques to extract a user or object assume, forexample, a green background that is easy to be distinguished, though notapplicable in applications where the green background is not available.

Once a model of the item or items to be manipulated is obtained, asecond problem is to create a smooth rendering flow that will update themodel and will render it smoothly back into the original video or intoother augmented reality video with the new information. The followingdisclosed embodiments achieve a smooth rendering flow relatively fastwith high quality, so the manipulation can be done in real-time, insemi-real-time, or relatively fast, depending on the requirement of theparticular application. Also, the embodiments are adapted to manipulatethe items (color, texture, shape) in real time, on camera main streamfeed, or off-line on user recorded videos.

Based on the model, the mask, and/or alpha channel per object from avideo, user movement is learned and different objects to fit or not tofit with the object shape and form can be installed in a convincingmanner. The disclosed embodiments can be distinguished from priortechniques, which attempt to do everything virtually. The embodimentscan include manipulation with a mask to cover other changes such asdifferent cloth, the changing face of a user, a change of the user'sbody type, elimination of a user from an image and the like.

Another disclosed feature is cloth recognition, which can identify theitems and the clothes the user is wearing. For example, it can include ahidden code or information in the fabric.

Other features and embodiments may include: a microwave scanner, wherethe microwave sensor is integrated in the screen; body measurementfeatures including, e.g., 2D and 3D body line detection or 2D/3Dmodel/mask generation out of a video; analytic conclusion from themeasurements, e.g., body type, origin, sex, age, suggested sizes ofclothes, suggested diet or treatment and the like; body line trackingover time for diagnosis, identification of changes, user identification,and the like.

In some embodiments, the main camera stream can be manipulated in realtime to change the appearance of the user, or to completely make himdisappear from the video or image. In this scenario, the camera DSP orseparate DSP or GPU capable device needs to get the camera stream andprocess the camera stream via remote control center and package it againas a camera main stream. A parallel computing to the camera main stream(sniffer) is also an option in one embodiment.

Provided below are several examples of additional applications that canleverage the technology of the present invention. Specifically, basevideo transformation can be used to create reliable virtual dressing.

Base video transformation can be used to create a reliable videoconference when the user/users on the other side of the line can see theparticipant as if the participant were looking directly at them, ratherthan having an image from the viewpoint of the camera. Alternately,transformation can be used to create an image as if a user were lookinginto the eyes of another user in order to provide a convincing, liveexperience.

Base video transformation for augmented reality can be provided formultiple purposes including though not limited to: body manipulation andresizing, body measurements, medical diagnostic/monitoring/trainingand/or tracking, gaming, dressing, dating, meetings, security, full bodysimulators, dancing training, hair dressing, makeup, general training,traveling, and the like.

The embodiments can be applied for video conference or gaming where theuser would like to change his appearance (e.g., skin complexion,dressing, face, and the like). According to the present invention, themodel can allow a smooth change into a desired appearance or even createan image whereby the user disappears from the video.

The embodiments can be used in a security application, with accurateface recognition over calibrated EyesMatch (see, U.S. application Ser.No. 13/843,001) when the camera is located with projection perspectiveto the user's eyes, with Dynamic QR (quick response) code recognition(e.g., authorization and management of user access via MobileApplication, or badge, QR code can be generated for the mobileapplication from the cloud and can be verified by user authentication inthe application), with full body recognition, scanning, and learning,with item recognition (e.g., clothes, uniform, badge, and the like),with a wireless sniffer (e.g., WiFi, Cellular, ultrasound and the like),with API to a security database, with location systems, with publicinformation and the like. The embodiments can be used with a threatprevention action based on recognition of a specified activity (e.g.,lock door, turn on/off a light, release fog, shooting machine, e.g.,electrical shock, bullets, paper and the like). Due to the geometry ofthe calibrated image, the location of the target in the image can bemeasured very accurately and can be used to operate an accurate attackon a target efficiently without risking others.

Real time and offline appearance registration and comparison can beprovided so a user can follow up appearance changes over time and overthe years. These processes can be provided for multiple purposesincluding medical registration, which can be over a given period oftime, between multiple examinations, using multiple/different sensors,e.g., for temperature, blood pressure, and the like.

In some embodiments, the calibrated camera/multiple cameras with/withoutcomplementary information from other sensors can be used for patientdiagnostic and patient monitoring. For example, though not limited, thepresent invention can be used to monitor and alert for skin problemslike skin cancer melanoma, breast changes that can suggest furthertesting for breast cancer, heart rate and blood pressure measurement,temperature of the patient, halo of the user, skin and finger nailrecording monitoring for changes, eye problems, spin, bones, muscles andbody flexibility trends and changes.

Augmented Reality Platform

FIG. 1 is a system block diagram for an augmented reality platformsupporting a real-time or recorded video/image processing. The systemcan include one or a plurality (1:n) of input devices 101, including avideo camera, a still camera, an IR camera, a 2D camera or a 3D camera.The input device 101 can be adapted to send information to one or moremachine vision augmented reality modules 102, 103, 104, 105, 107, 108and 109. The one or more machine vision augmented reality modules 102,103, 104, 105, 107, 108 and 109 can be adapted to send information toone or a plurality (1:m) of screens 106. The one or more machine visionaugmented reality modules 102, 103, 104, 105, 107, 108 and 109 can beadapted to send/receive information to/from an interface or userinterface module 110. The interface 110 can be adapted to send/receiveinformation to/from one or more of a cloud, a web/store or a userdevice, e.g., smart phone or tablet.

The one or more machine vision augmented reality modules 102, 103, 104,105, 107, 108 and 109 can include an image grabbing module 102, aneyes-match transformation module 103, an augmented reality module 104, avideo/still recording module 105, a trigger event module 107, a controlelement module 108, and a factory calibration module 109.

The image grabbing module 102 can include one or more of the followingfeatures: enhancement filters, format conversion, video frameseparation, image cropping, image resizing, image stitching and thelike. The image grabbing module 102 can be adapted to send informationto the eyes-match transformation module 103. The image grabbing module102 can be adapted to send/receive information to/from the trigger eventmodule 107.

The eyes-match transformation module 103 can be adapted to apply on theimage the right mapping to match the camera point of view withtheoretical mirror point of view (user eyes reflection) and fill theblind pixels if there are any after the mapping. The eyes-matchtransformation module 103 can be adapted to send information to theaugmented reality module 104 and/or the video/still recording module105. Also, the eyes-match transformation module 103 can be adapted tosend/receive information to/from the control element module 108.Further, the eyes-match transformation module 103 can be adapted to sendinformation to the one or plurality of screens 106, to display an imagethat mimics a reflection of a mirror.

The augmented reality module 104 can be adapted, for example, to providevirtual color and texture replacement, virtual dressing, objectinsertion and the like. The augmented reality module 104 can be adaptedto send/receive information to/from the control element module 108and/or the video/still recording module 105. Also, the augmented realitymodule 104 can be adapted to send information to the one or plurality ofscreens 106 to display an image that mimics a reflection of a mirrorthat is modified by the changes produced by the augmented reality module104.

The video/still recording module 105 can be adapted to record a singleimage (frame) or a short take based on software control. The video/stillrecording module 105 can be adapted to send/receive information to/fromthe control element module 108. Also, the video/still recording module105 can be adapted to send information to the one or plurality ofscreens 106.

The trigger event module 107 can include one or more of the followingfeatures: recognition of a user in front of the mirror, facerecognition, recognition of user gesture commands, item recognition,distance measurement, user body measurements/assessments (including,e.g., height, age, weight, ethnic group, sex, and the like) andcalculation of user theoretical point of view in theoretical mirror. Thetrigger event module 107 can be adapted to send/receive informationto/from the control element module 108.

The control element module 108 can include one or more of the followingfeatures: control and management for setting the camera to optimizequality, for setting other hardware elements, an interface betweenalgorithm modules and higher code/application/user interfaces, and pushcalibrated data from factory into the algorithm elements. The controlelement module can be adapted to send/receive information to/from thefactory calibration module 109.

The factory calibration module 109 can be adapted to define the mappingtransformation between the camera and the user point of view in front ofthe screen. Also, factory calibration module 109 can be adapted tocalibrate the image based on distance, a special location (e.g.,markings on the floor), user height or any combination thereof.

FIG. 1 and the following description represents just example of oneembodiment of the present invention; other flows or functionalities canbe allocated between the modules, represent additional embodiments thatare part of the invention. The present inventors propose two methods toenable the augmented reality capabilities (real-time and offline). Bothmethods wrap the augmented reality module 104 with real image or videodata that is real-time or was taken post processing via, e.g.,eyes-match transformation module 103. That is, the image that isoperated upon by the augmented reality module 104 may be a transformedimage that represents a mirror reflection of a user.

One feature is that a user can define manually or automatically (via,e.g., interface 110) rules regarding what items the user would like toprocess and manipulate and what is the expected end result, e.g., anautomated rule can be something like automatic identification of auser's shirt, which can be automatically or manually changed to adifferent color, or a user selecting a shirt via manual selection with atouch screen and manual color selection. Then, the selected object canbe processed and extracted/segmented out and recorded to the databaselinked to the original recorded video or image. The augmented realitymodule 104 can then process the model/mask in real-time at a given framerate, which can be at a lower or higher frame rate than the original,and at the same or a different size than the original. Some applicationsdo not require (but can nonetheless include) live augmented reality,like trying-on clothes when the user would like to see himself with themodification (one or more options). Once the extracted object from thelive scene is saved, it is easier to render multiple changes (color,texture, size, and the like). In addition, it is easier to perform alonger process, much more accurately, with higher quality and utilizinga process that yields more information, e.g., user movement, bodymeasurements, and quality based on frame integration and the like. Forother applications that require real-time processes, like a videoconference, the second option is better to use. With the second option,the performance should be adapted to be as close as possible to theactual frame rate of the camera in order to avoid introduction ofdelay/lagging that can yield frame drops and poor performances. Ifdelays/lagging are encountered, then such delays/lagging can be managedin a way that yields a smooth video without lagging of the frames. Suchmanagement can be achieved with buffers, image timestamps plus somedelay, or with timestamps and frame reschedules for rendering.

For video input, it is highly recommended that the rendering process bedone in a DSP or GPU device in order to avoid introduction of delay intothe video. The creation of the mask/model can be done for the firstoption (not real-time) in a CPU as well. For the second option, themask/model can be performed and calculated in the GPU/DSP as well.

In the trigger event module 107, some of the trigger functionality canbe completely automated, e.g., a process can be started if facedetection or presence detection is made. Some of the triggers can beperformed in a semi-automated fashion from the user interface module110, which can include any way to control the computerized device. Someof the functionality of the trigger event is to calculate the imagetransformation based on geometric information, calibration, and/orreal-time user tracking, e.g., location of the user, eyes, head, hands,position, movement and the like. The tracking can be done using one ormore techniques such as background substation, pattern recognition,color segmentation, and the like. The transformation trackingcalculation functionality can also be implemented in the other modules.

The control element module 108 can be adapted to configure the systemsetup, camera device, authentication and the like, and can also provideinformation from the tracking transformation function into actualgeometry transformation module or augmented reality module and the like.

With the factory calibration module 109, some of the information neededto calculate the transformation to be applied on the image/video can begenerated during factory calibration or can be calculated based onadditional information on the actual orientation of the camera in thefield, e.g., height above the floor or desk and the like, 3D point ofview, lens field of view (FOV), and the like. The factory informationplus the actual geometry of the implementation can be processed anddelivered to the right element in the system that will use theinformation for better calibration and accuracy.

In addition to application to a local screen for a mirror application,the present invention can be used to send the video plus the means totransform it or the actual manipulated and transformed video to anotheraddress in the cloud or locally as a single or bidirectional videoconference.

FIG. 2 depicts an example of an augmented reality module, which cancorrespond with the augmented reality module 104 described above.Specifically, the augmented reality module can have a function ofallowing a user to virtually dress themselves, change appearances, suchas color, accessories, etc. In this embodiment, the system obtains inputimage or video from, for example, the EyesMatch computerized method 201,or from any other image/video source, e.g., user smartphone, securitycamera, Google glass, mobile camera or stationary camera. Additionalembodiments can include additional geometric information that will helpto calculate proportion like user height, gaze and the like. If the uservideo or image is coming from the EyesMatch module (calibratedimage/video), a more comprehensive model can be created that allows forbody measurements, object pose, size, highly accurate orientationdetection and the like. The additional information that can becalculated from the calibrated object or video can allow for objectfitting, object replacement and insertion of new objects into theframe/video, since any distortion introduced by the location and fieldof view of the camera has been accounted for and corrected. Thesecorrections enable highly accurate measurements of the user height,waist, etc., and fitting of the user's body to generally classified bodytypes.

The election module 202 can obtain election information from interface206, manually from the user (X,Y or name of object) or automaticallyfrom an electing method, e.g., a mechanism that can automatically detectpredefined objects like pants, dress shirts and the like. For example,the interface 206 may enable the user to select a garment to be changed,e.g., change color of fabric type. This election is sent to the electionmodule 202, so that the proper item to be modified is extracted from therest of the image.

Extraction of the object is a rather difficult task. For example, if theuser is wearing a red shirt, and in the background there's a red chairand the user is standing on a red carpet, the human eye can easilydistinguish which red parts belong to the shirt and which do not.However, performing such determination automatically with a computer isvery difficult task. Moreover, depending on the lighting and the foldsof the shirt, different pixels belonging to the shirt will appear asdifferent colors. Therefore, if one goes by simply choosing all redpixels, some dark pixels belonging to the shirt will not be included,while some pixels belonging to the chair and carper will be selected.

The extraction module 203 can determine the location of the object(i.e., all of the pixels that belong to the object) and samples thecolor (or the average color of the object, which can be more than onecolor). The module 203 can use this information to create a black andwhite mask that is first used to generate a 2D or 3D textured shaded andcolored mask.

The technique to extract the object is based on 3D color correlation orany other technique such as the closest Euclidian distance between theobject average color and the pixel color to separate the pixels of theobject from the entire image. The decision as to whether the pixel is inthe object or not can be performed in multiple levels and is not limitedto the following examples:

1. Color correlation and the first decision can be based on Euclidiandistance threshold, where the Euclidean distance threshold is in the RGBcolor space or chromatic color space. That is, for each pixel, aEuclidian distance is measured from the pixel's color to the objectcolor as determined by taking a single or average color of the object.If the pixel's color is at a distance higher than the threshold, it isdetermined not to belong to the object.

2. Filter noise by applying morphological operators like dilution anderosion, which can improve the decision regarding pixels that are“wrongly labeled” as part or are not part of the object. In mostimplementations, the shape of the object would be known or may beestimated. For example, if the user selects pants, the system would havean estimation of the shape of pants. Thus, the results of the selectionin (1) above can be further adjusted by checking whether each excludedpixel is within the estimated shape and should be included in the objectand, conversely, whether an included pixel is actually outside of theestimated shape of the pants and should be excluded.

3. Decision based on information from previous or forward frames, orfrom neighboring pixels in a row or around the pixel. This steprepresents a major decision in the process. In most implementation, itwould be unusual for consecutive pixels to be alternatingly selected asbelonging and not belonging to the object. Physical object are largerand are covered by more than a single pixel. Therefore, the pixels wouldbe grouped as belonging or not, and a check of neighboring pixels can beused to confirm the selection.

4. Object distance from original election, which can be used as athreshold. For example, for each pixel its Cartesian distance to theoriginal selected pixel is calculated and, if it is outside of theexpected distance, the pixel is considered to be outside of the object,and vice versa.

5. Object surface continuation, whereby, if we know that an objectsurface continues, then we can filter out some of the noise. Forexample, sometimes the image may have reflections or shadowing thatwould cause the pixel color to appear drastically different from thetrue color and therefore, to be indicated as not belonging to theobject. However, an object is generally a continuous surface. Thus, thecontinuity can be used to eliminate such mislabeling.

6. Object edges, whereby we can improve the decision around edges byedge detection that can be done by high pass (HP) filters or othertechniques.

7. Decision based on color energy. One of the problems of colorseparation is that color under low light conditions can be seen asblack, and the dynamic range of the decision is reduced significantly.Dark/black pixels can be isolated and other techniques can be applied todecide if the dark/black pixels belong to the object or not, e.g., thepresent invention can determine if the pixel is located inside theobject boundary, or the distance of the energy from the object standarddeviation (STD) color changes.

8. Use previous information on the expected object shape to obtainbetter results.

9. In case the object is a combination of multiple colors or shapes orhas logos or other pictures, multiple color correlation and combiningcan be used. Additionally, any of the multilevel methods specified abovecan be used to obtain a higher level decision regarding the object.

10. The decision can also be based on a majority or a decisionpertaining to a neighboring pixel/image as a weighted factor in thedecision. In case we are processing the decision on the image as avector, it can be easier to look at the neighbors in the same row orsame column depending on how the image matrix is reshaped into a vector.

11. Estimation of the item material/texture color standard deviation(STD) can also add significant information for object segmentation. Insome implementation, a database of different fabrics and their texturedis stored to assist in such determination.

12. Any combination of one or more of steps 1-11 above.

The mask can be used for rendering as a simple black and white mask.However, in order to create a convincing feeling of an object,additional information from the object's texture or appearance can bemaintained. In order to obtain the additional important information, themask can be applied on the original frame or video, and the RGB or grayscale texture, shade, or brightness scale on the object can be obtained.This information is much more accurate and convincing for color changessince it saves the original object's wrinkle texture, shading, lightreflection, material signature, and the like.

The model mask can be constructed in layers for improved manipulation.Examples of a potential layer structure can be as follows:

1. Black and white mask (to segment out the object). The black and whitemask can be very important to distinguish between the object and thebackground or between the object and another element around the object.Multiple techniques can be used to optimize the object mask/boundariesdecision.

2. Object edge mask—representing the edge or outline of the object.

3. Red color mask—representing red areas of the object.

4. Green color mask—representing green areas of the object.

5. Blue color mask—representing blue areas of the object.

6. Textures that apply to all color masks—representing textureappearance of the object.

7. Shade or brightness mask—representing shaded or bright areas of theobject. This may be an intensity map of all of the pixels belonging tothe object.

8. Material light reflection mask—representing light reflection of theobject.

9. Material light absorption mask—representing light absorbing areas ofthe object.

10. Mask from other sensors like IR, microwave, depth, ultrasound, ultraband and the like.

11. Layers similar to those described above.

Once the mask model has the required information, in order to changecolor or texture, the rendering module 204 can be used to modify thespecific layer/layers and regenerate the object out of the multiplelayers resulting in a rendered video 205 that is extremely realistic.For example, if the user wore a red shirt, the red mask can be replacedwith a blue mask to render the user wearing a blue shirt. Since allother masks remain the same, the blue shirt will be rendered with all ofthe shading, brightness, reflection, texture, etc., of the red shirt,thus rendering a very realistic blue shirt having the same folds andfollowing the same body contours as the red shirt. The effect of somelayers can be introduced by multiplication or by adding the modifiedlayer to the frame. Subtraction and division can also definerelationships between layers. Additional techniques that allow for morecomplex item manipulation include a registration technique, which can bebased on a few points stretch/transform an object or material to fitwithin the boundaries of the manipulated object.

The module can obtain the original video/image, the modeled maskmultichannel and the required change. The required change can be anycombination of color, brightness, material, texture, icon/logo,printing, fabric and the like.

In one embodiment, the required change can be outside or inside theoriginal object boundaries and a modified mask for the new objectboundaries can be created to replace the original mask model.

In one embodiment, the required change is fabric with specific orientedtexture and color that can be inserted in a specific orientation. Inthis embodiment, the material orientation can be modified and appliedaccordingly. For example, a database can store masks corresponding todifferent fabrics, such that a user can change the fabric of an item theuser is wearing by simply selecting a different fabric on the userinterface. The system would then replace the mask corresponding to theactual item the user is wearing, with a fabric mask corresponding to theselected fabric.

In one embodiment, the required change can be another user's try-onmask. Using a registration technique, the other user's try-on mask canbe applied to fit the new object on the user's similar object to try-on.For example, a user is measured for a shirt and another user is measuredfor a jacket. In order to show the first user how the jacket from thesecond user looks, the method can use a registration technique that fitsthe second user's jacket so as to obtain and apply the correct body lineshape of the first user, which can create a more convincing fitting. Onebenefit of this method is that the jacket will look much more real inthat it is rendered so as to preserve all the texture of the seconduser. To produce these results, the method can register the orientation,location and size of the first user. Registration point may include, forexample, midpoint between the eyes, the edges of the shoulders, theedges of the waist, the knees, etc. By stretching or contractingregistration point from one user to another user, the garment from oneuser can be shown as if worn by another user.

In one embodiment, the mask can be used as pointers for virtual objectregistration. In one embodiment, the mask plus the background recordingin the right transformation as the user can be used to eliminate theuser/item from the image/video.

In one embodiment, election of the single object or multiple (1:n)objects can be obtained that need to be modeled. From the video, a maskis created per frame. If the user is spinning, a 3D or partially 3Dframe by frame model can be created. From this frame by frame model,different perspectives can be obtained and used to create a 3D modelthat includes some or all of the user's movements. Later on, thisinformation can be used to create a more convincing virtual dressing.That is, the present method can utilize the user's own movements informing the model.

In one embodiment, the rendering can be performed in the GPU, CPU, cloudGPU or cloud CPU. The input elements to be rendered can come from theCPU, from the user database in the cloud, or from an active link withthe inventory/any other database/3D printing, ecommerce data base,social data base and the like.

Changes in item color can be based on actual try-on and availableinventory or optional 3D printing on site or in the factory for usercustomization. One advantage relative to regular virtual dressing isthat the item is shown draped on the body just as it would in reality,since all of the folds, shading, etc., will be transferred with theappropriate masks. This is a very important feature from many aspects.The user can sense how the item feels on his body, how it may effect andchange his body shape and the like.

In one embodiment, an accessory or any other item can be added bylearning the dynamic movement and the mask model of the relevant object.Also, the background can be augmented so as to change or create adifferent environment by the same technique. Once all the requiredobjects are labeled, the required objects can be masked and the combinedmask can be used to change the background.

In one embodiment, the rendering module can render the object with anenhanced rendering technique of interpolating the object and the frameinto higher resolution, can combine the object at high resolution, cansmooth the edges, and can decimate the object back to the requiredresolution with better quality of integration into the frame. Additionaltechniques include working directly on the edge of the object byaveraging with some weighting factor the value of the pixel in order tobetter blend the object with the background color.

FIG. 3 depicts an augmented reality module that can replace a body part,color, orientation and texture of an item or object in the foreground orbackground of the image, e.g., the module can add hair to the user,change the user's eyes, skin and hair color, can change the pose of theeyes and the like.

Modules 301, 302, 303 and 306 can work in a similar manner as comparedto modules 201, 202, 203 and 206 of the previously described generalaugmented reality module. Module 304 can have the capability ofcalculating or obtaining additional information like head pose or bodydirection of movement directly from the EyesMatch module or from module307 utilizing a dedicated detector for item 3D movement and orientationand can use this information to modify the required body part, e.g.,obtaining the head pose will allow for eyes steering correction bymodifying the eyes of the mask/model to the required direction. Inaddition, the head detection can allow for adding hair in the rightorientation, a hat and the like.

In one embodiment where the required mask in some dimensions is smallerthan the original mask, module 307 can perform or deliver in real-timeor offline EyesMatch transformation of the recorded background image orvideo. The background transformed image or video can be capable ofrendering a part of the body part or item using foreground or backgroundmanipulation, e.g., in case one would like to eliminate a user's head orupper body from the video. The transformed background image or video canbe applied on the mask that captures the head and upper body, and theresult can be rendered with the original transformed or non-transformedvideo. The result can be a rendered video 305 without the user head andupper body.

For example, in a more complex case, one might wish to depict a shorterlength of a given dress. The mask manipulation in module 304 can berequired to create a shorter mask for the new dress, and the differencebetween the original mask and the mask after manipulation can be a newmask for manipulation. In the new mask, some part will be the estimationof the user's legs and some part will represent background that would benewly visible with the shorter dress length. The new mask can be dividedto legs and background, and the new rendered object can take thecombination of background image and predicted legs to create a newrendered item. The result after rendering the modified item into thevideo is a user with shorter dress. Multiple techniques can be used topredict how the legs should look in the useable area.

Colograma

In order to manipulate color or appearance of an element in the frame,the pixels belonging to that element needs to identified. This isnormally done by comparing colors and assuming that pixels of same colorbelong to the same object. However, such a procedure is not accurate. Inthe disclosed embodiments, the comparison is done using variables otherthan color. The method attempts to simulate the human eye, which candistinguish objects even when parts of the objects may be shaded andother parts may be highly illuminated or even reflective. For example,in one embodiment the object is recognized as a known object withpredictable boundaries.

When implemented as virtual mirror, the objects may include clothingitems, such as shirt, pants, jacket, etc. So, when a user selects ashirt for manipulation, the system utilizes a predictive association topredict the location of the pixels that should be included in theobject. For example, all pixels above the head of the user and allpixels below the waist of the user should not be related to the shirt.Thus, the search space is reduced. This predictive algorithm is improvedwhen it is applied to a frame which undergone the calibration procedureof the EyesMatch. To provide a striking example of the combination ofthese methods, if the user elects to change only the color of the user'sshoes, when the frame is calibrated using EyesMatch, the approximatelocation of the shoes is known, and all pixels above the feet of theuser may be ignored when searching for the pixels belonging to theshoes. This drastically limits the search field.

In summary, in order to identify the object for manipulation, inaddition to color comparison, the pixels may also be evaluated using thepredictive location of the item, the known or approximate geometry ofthe object, the texture of the object, a standard deviation from thetexture, standard deviation from a representative color, etc. Each ofthese variables can be assigned specific weight in a statisticalevaluation of each pixel. Also, each parameter may be assigned adifferent error or statistical deviation value.

FIG. 4 depicts computing methods to create a model for color and textureexchange, which can be referred to as a Colograma. This technique ismost beneficial for parallel computing that can support a large numberof users or a large number of frames/video as opposed to super highquality color exchange techniques that can be found in software programssuch as Photoshop. Such methods can take a lot of time and may not bepractical to conduct on any large number of user images or video. Thedescription of FIG. 4 is just an example and any derivative of thedepicted flow is part of the present invention.

One challenge to change a color of an object in a video or image is toaccurately identify the relevant pixels of the object. In a video file,speed is a limiting factor for applicable transformation. In FIG. 4, asimplified example of a method to segment/extract an object from a videois depicted. The method of FIG. 4 may be executed by the systemsdescribed with respect to FIGS. 2 and 3.

The image or video to be modified is received in 401. In 402, the frameof the color image or video is converted into a line vector, which isoptional, although vectorization of the image can speed the process timedramatically. The size of the vector is 1 by n, wherein n is the totalnumber of pixels in the frame, i.e., the total number of pixels in thewidth times the total number of pixels in the height of the frame, time3 for each of RGB colors. Also, in 403, the effect of the brightness iseliminated. There are many techniques to eliminate the effect of thebrightness. In this example, energy averaging per pixel in the XYZchromatic space is used, by dividing each pixel by the sum of XYZ. TheCIE XYZ color space encompasses all color sensations that an averageperson can experience. Y is defined as the luminance, and for any givenY value, the XZ plane will contain all possible chromaticities at thatluminance. For example, a 3×3 matrix can be used to convert RGB to XYZ,using the chromaticity coordinates of an RGB system (x_(r), y_(r)),(x_(g), y_(g)) and (x_(b), y_(b)) and its reference white (X_(W), Y_(W),Z_(W)).

In parallel, at 404 the object selection is performed by firstdetermining a comparison color that can be used for examining all pixelsfor determination of whether each pixel does or does not belong to theobject. The comparison color is determined by using statistical analysisof the color of several K pixels that are believed to be within theobject. This may be done by first selecting several points K(x,y)believed to belonging to the object to be transformed, by, e.g., knowingthe geometry or the approximate location of the object within the image.K is the number of locations/zones with distinguished color that can besegmented out of the background or out of other objects. In someimplementation, for each location, a window, or zone, is selected aroundthe specific point, so as to ensure that the point or pixel is not ananomaly, but rather representative of the particular zone. Then at 405each point K(x,y) is passed through the same transformation as performedin module 403. At 406, k iterations are performed to find each pixel'scolor and from that to find the best representative color of the object.This color can be used for the follow up investigation to find all ofthe pixels that belong to the object. K equals to or is larger than 2 inthis technique. For each k, the Euclidian distance 2D or 3D iscalculated. The minimum distance and K value is saved and is used as thecomparison or representative color. This operation can be done on allpixels at once in a relatively fast process.dist=sqrt((X−xi(k))²+(Y−yi(k))²+(Z−zi(k))²)

After K iterations, the labeled image can be obtained. The Euclidiandistance “dist” is just an example of a computational method todistinguish between colors; there are other methods to calculate thedistance between colors, e.g., a color distance model based on humanperception of color (chromatic, saturation and brightness), advancedcalibrated techniques to match the sensitivity and the ability toseparate color with the human eye like in CIE76, CIE94, CIEDE2000 andthe like or any combination with histogram stretching IR/3D depthcamera, color integration over time or any other method to improvesensitivity of the color detection (module 411). Applying or crossingthe additional information from module 411 can happen at the distancecomparison level 406, at the very end of the model creation 409, or anycombination depending on the nature of the additional information(deterministic, statistic, time variant, and the like).

For example, when an IR camera is used, the image of the IR camera canbe registered with the image of the standard camera. Then, the image ofthe IR camera can be used to easily separate the user from thebackground. This information can be used to limit the pixels of thestandard camera that are to be evaluated whether they belong to theobject or not. For example, when the comparison is done using a vector,the information from the IR camera can be used to reduce the number ofentries in the vector that should be evaluated. Similarly, a texturemask can be used to remove pixels from the evaluation vector. Forexample, the texture of the carpet can be stored and any section of theimage having the same texture can be eliminated from the evaluationsince it is known that the user would not be wearing a carpet. Thus,several layers or masks may be used to operate on the vector andincrease the accuracy of the results of isolating the object andassigning all of the pixels that belong to the object.

In addition to color difference we can also use other techniques thatcan add information about the object to improve the decision such as:zone likelihood (a given pixel needs to have neighbors or some mass ofpixels), zone characteristic, boundary filters to isolate the objectboundary before making a final decision, depth information (whichtypically needs to match the contour of the depth information with theend image of the object in 2D or 3D), time integration to determine ifpixel is in the object zone over multiple frames, and the like.

Module 407 is an example of one embodiment of how to distinguish betweenthe required colors and the other color space. In module 407, all thepixels with a distance greater than a threshold are zeroed out as nonrelevant (a pixel with a color different from any one of the colors 1 tok), and 1 is assigned to all relevant pixels, thereby generating abinary mask which identifies all of the pixels belonging to the object.

Module 407 is an example where it is desired to separate a specificcolor or colors. Here, all the indexes can be zeroed out except therequired one. The process proceeds as follows: zero out all thenon-relevant indexes, obtain a background and non-relevant colorsvalue=0, and elect the required color object labeled=1. If there is morethan one color in the object, 1 can be assigned to any elected index 2to k+1 and zero to all the others.

In 408 a black and white filter can be used to clean noise and smoothout the shape of the object. Other techniques may be used to improve thedecision of which pixels belong to the object. As a result, an index forall the relevant colors starts at 2 to K+1.

At module 409, the obtained black and white mask is applied on theoriginal color image and the 3D model for color and texture changes isobtained. The model can be 2D alpha channel of gray scale or 3D in thecolor space. Module 410 can obtain a 2D or 3D model of the object. Incase of video from a single camera, it is possible to obtain a 3D modeleven if the user is moving in front of the camera, e.g., turning aroundin front of the camera. In this case, it is also possible to obtainobject measurement in multiple cuts to estimate the user's 3D bodycurves.

An example of a 2D input to the model generator is provided below, whereit is desired to create a model of the user's blue shirt shown in FIG.9. FIG. 10 is an example of a 2D model or mask of the shirt, without thecolor information. Instead, a grey scale mask of the selected object, inthis case the shirt, is generated and can be used later with any appliedcolor. The texture of the shirt is preserved in this manner, so it isrelatively easy to manipulate the color or the texture or even changethe boundary of the model to create a different object.

The model based on just color difference is not perfect in terms ofquality hence additional information and techniques can be used toimprove the quality of the object model (see, module 411). Additionalinformation techniques like interpolation and decimation or edgesmoothing can be apply after processing via module 410 in order toimprove the quality of the model.

Body Measurements

FIG. 5 depicts an example of a method to extract a body measurement froman image, multiple images or a short video. The body measurements can beused for various applications and services, such as, e.g., estimation ofweight, determine best fitting clothes, monitor body shape and weightover time, etc. Since the system acquires many images of the user atdifferent stances and orientations, it may be beneficial to select thebest image for performing the body measurements. The determination ofwhich image to select may be performed before or as a part of theprocess illustrated in FIG. 5. Specifically, a good body measurement canbe obtained when the image is of user's body in perpendicular alignmentto the optical axis of the camera. This may be determined by firstdetermining whether the face appear symmetrical along a vertical linepassing midway between the eyes. If the face appears symmetrical alongthis line, a vertical line along the center of the body can be used toconfirm that the body is symmetrical along that line. For this step, itmay be beneficial to first remove the hands from the image, such thatonly the center of the body is considered from the symmetricaldetermination. The process of removing the hands from the image isdescribed more fully below.

In module 501, the image is analyzed using a body line and Body MassIndicator (BMI) analysis by first using a process similar to theEyesMatch approach described in U.S. application Ser. No. 13/843,001. Inthis approach, the image can be rearranged and transformed to compensatefor distortion due to the setup geometry of the camera and opticdistortion. Example of setup distortion factors include the userdistance to the camera, the camera height, camera projection angle, theoptics of the camera's FOV and other optic distortions. After the imagecorrection, each pixel area representation can be roughly uniform inarea (cm²). Processing can be done in CPU, GPU, DSP in the camera, localcomputer near the camera, or remote server.

Module 502 is an optional element, which can transform a 2D image to avector conversion to speed up processing. The size of the vector is theimage's width, times the height, time three pixels (for RGB). The imageinputted into module 501 can also be a vector already. It can be easierto perform some types of image processing on 2D matrix while other typesof image processing are better suited for the vector image.

In module 503, it can be easier to perform the color manipulation toeliminate light and saturation effect on the vector image. This is donein a similar manner to what is described with respect to 403 above.

In module 506, the image can be made to pass an edge detection filterthat can emphasize color changes and enhance the visibility of the bodyline. The edge detection can support edges of the body line so it cansupport multiple edge directions. For example, an edge filter withvertical, horizontal and +/−45° capabilities can yield a good edgedetection of the body line.

Modules 504 and 505 can provide additional information into thealgorithm that can help with the image analysis, e.g., sex, age, raceand the like of the user and statistical proportions associated with theinformation. This information can be used later on to focus the searchof body parts. The height measurement can be used with complementarybody proportion information to focus the search of specific body parts,for example, the bust, navel, waist, hips and the like.

Module 507 can provide head detection. There are multiple techniques forhead detection. For example, one can take the edge emphasized image,flip it along the body length, and perform correlation between theimages. The peak of the correlation can indicate the central mass of thebody. Another alternative method is centroid center mass calculation ofonly the edges. Centroid center mass calculation can be less accurate ifthe light on the user is not uniform enough; however, this technique canbe faster. Other techniques can be based on pattern recognition, eye,nose, and shape detection of the head. Once the central mass is found, arelevant window can be cropped to identify the body from the image.Also, the emphasized edge of the tip of head can yield the height of theuser in pixels. Since the image has a uniform weight of length perpixel, the total height can be calculated accordingly. The assumption isthat the module 501 transformation will yield consistent manipulatedimages across arrangements, e.g., 1-5 meters (m), where the actualheight of the user on the screen in pixels will be proportional to theactual user height. Additional factors can be added if this assumptionis not exactly the case per all distances.

Module 508 can measure the size of body parts in pixels. The techniquecan be an additional process on a focus area to further emphasize thebody part. FIG. 13 depicts an example of the result of additionalprocesses that eliminate the arms and hand from the body and point todifferent intervals of where to look for particular body measurements.The intervals can be calculated by generating a vertical centerline,1301, which may pass through the center of gravity or the axis ofvertical symmetry of the user's image. Then horizontal section lines canbe added to generate areas fit for body measurements. For example, line1303 delineates the location where the legs connect to the torso, whilelines 1304 and 1305 delineate the narrowest area of the waist.

Measurements in pixels can be translated into centimeters (cm) accordingto the predicted curve, e.g., the neck can be modeled as a cylinder sothe measured width of the neck in pixels can represent the 2×radials inpixels, then the radials in pixels can be converted into cm per thecalibration. The bust for a man has a more elliptical model, so thetranslation will be a bit different. If additional side measurements areavailable, such measurements can be added to provide more accurateinformation to the model.

In module 510, the measurement can be deduced from the body type of theuser, e.g., a “pear” shape if the hips are wider than the bust and waist(see, FIGS. 11 and 12). In addition, some analysis can suggest thefollowing:

1. body condition health risk for heart issue and the like;

2. the type of clothing that best fits the user's body type;

3. trends in the user's body for historical comparison;

4. focused advertising based on the user's body type (e.g.,Mesomorph/Endomorph can get focused coupons for the best nutrition thatfits their body type);

5. body recognition; and

6. body diagnosis can monitor body changes over time (spin condition,flexibility, potential tumors, and the like).

Information on the user's BMI can be learned or estimated from theobject model described in FIG. 4 or directly from the image BMI analysisas described in FIG. 5.

In various applications it is beneficial to “learn” the texture ofspecific articles in the image. According to various embodiments, tomeasure texture with a good resolution, the object is first isolatedfrom the image. In one embodiment, after segmenting out an element orbody part, additional fine texture reconstruction or measurement can beimplemented. Texture measurement can be done with 2D, or 3D sensor(camera).

In embodiments utilizing a 2D sensor, the following two examples can beimplemented. Since the user will be moving in front of the mirror, thesystem can capture two images (e.g., consecutive images) that will beused to calculate depth or texture. There are multiple ways to do soand, as an example, the system can extract two or more images from thevideo and based on central mass detection can calculate the distance anduse the distance for the stereoscopic calculation of the texture. Thatis, when taking a stereoscopic image using two cameras, the distancebetween the cameras is known, so that triangulation can be performedusing the distance between the two cameras and their angle of opticalaxis. But if there's only one camera, then the system takes two imagesusing the same camera, but separate in time. Consequently, the userwould have moved a bit between the two images. The system thencalculates the difference between the two pictures and performed reversetriangulation. This can be done by calculating, for example, centralmass of each pictures and then calculate the x−y distance between thetwo central masses (for example, in pixels). This can be used to performreverse triangulation as if the two images were taken using two cameras.Other pointers can be used to calculate the distance, e.g., theprocessor can identify the eyes in the two pictures and calculate thedistance between the eyes in the two pictures. Another option is todetermine rotation of the user, i.e., (r,θ) and use that to perform thereverse triangulation. In any of these methods, since the camera can bepositioned at a height that is above the central mass of the user, e.g.,on top of the monitor, the reverse triangulation can be used tocalculate depth, thereby acting as a depth sensor.

Alternatively, the system can find noticeable pointers in both imagesand try to match them, then eliminate all the pointers without clearidentification. This can be done using, e.g., RANSAM Random SampleMatching technique to find from a random set of pointers the group thathas similar behavior and use those pointers to find the distance inpixels for texture measurement or 3D construction of an element.

Another option is to estimate the distance between the mirror based onthe distance and the body rotation. For example, if the system canmeasure the user's body pose, it can estimate the texture.

With 3D sensor, the system can do the same as in 2D, although in orderto improve the dynamic range and the accuracy the system needs toisolate the element to be measured. The background subtraction can beimproved by rough depth measurement threshold. The system can estimatethe texture every 1 or more frames and just smooth the texture changesbetween frames by Kalman filter or any other interpolation extrapolationtechniques.

In some embodiments, only a 3D sensor will generate depth measurementthat will be used to control a parametric model of the user's avatar.The user can be virtually dressed and explored in 360°. In addition thesystem can be controlled to adapt position movement to the user formirror application or video conferencing. To match the avatar to theuser movement, a registration technique can be used where theregistration pointers can be allocated in multiple ways, e.g., automaticnoticeable element detection and RANSAM random sampling and matchingtechnique. Once the system has the pointers, it is easier to move theavatar based on the actual user movements. The data from a depth sensoror 3D camera can pass a similar mapping transformation engine and theresult can be the base of a more accurate 3D avatar of the user. Theavatar can be used fully or partially as a model base to present virtualelements on the user.

Other technique to fit the registration mapping is to identify elementin the image like head, hands, upper body legs and match the movement ofthe avatar to these elements motion. The motion can be measured andestimated every 1 or more frames and the smooth coarse of the avatarmovement can be done by Kalman filter technique. The avatar motion canbe done in real-time, semi real-time, or offline.

In some embodiments, the system receives the stream of images (eitherfrom one camera, from two cameras, or from a 3D camera), and use thestream of images to generate virtual reality in real time. The originalimages from the camera are discarded, and instead the virtual world ispresented on the monitor. Since the real time images are used togenerate the virtual reality, the virtual reality images faithfullyrepresent the real world as imaged by the cameras. Thus, as noted above,the real time images can be used to generate an avatar in real time,where the avatar faithfully represents the user captured in the images.On the other hand, the background and articles worn by the user can bechanged, such that the user's avatar can be displayed as if it is in adifferent location wearing the same or different articles as the user.For example, if the user tries on a ski coat, the real time images canbe used to generate an avatar that corresponds to the user and whichwears the same coat, however the background can be changed to display aski slope, so that the user can imaging himself wearing the coat on aski slope.

On the other hand, it is also possible the store the real time images,and then operate on the images off line, such that higher processingcapacity is available, so as to generate a movie using avatar based onthe user. The movie can then be sent to the user, e.g., over theInternet.

Security and Other Applications

FIG. 6 is an example of a security system application for multileveluser learning and authentication. This is just one example of possibleflow implementation; any combination of the flow or different functionalseparation between the diagram modules is part of the invention.

As in the previous figures, stills or video from device 601 can be inputinto an image grabbing module 602. The image grabbing module 602 can becontrolled by a trigger event such as those previously described andprocessed by trigger event module 607, which can provide trackinginformation and rules to initiate an image grabbing process.Specifically, the image grabbing module 602 can grab the image from theinput 601 and push it into the EyesMatch transformation module 603 withthe additional tracking information (real time geometric measurement).Based on the tracking information and the event requirement, theEyesMatch transformation module 603 can calculate a transformation tomanipulate with the camera point of view, angle of view, and the like tocreate a calibrated image of the scene. In addition, the EyesMatchtransformation can be performed locally in GPU/CPU/DSP, in the cloud andthe like. In one embodiment, advanced information on the user or on thegeometric structure of the scene can feed the tracking algorithm and maytake part in the calibration. The additional information can be providedas one time geometric information, e.g., user height, distance betweenhis eyes and the like. In that case the additional geometric informationcan be used to calculate the required transformation. The differencebetween the distorted image and the known geometric can be used tocalculate and calibrate the transformation.

The calibrated output from the EyesMatch transformation module 603 canthen be fed into the security scanner module 604, which can also becalled a scene analytic module. The security scanner module 604 can beadapted to learn one or more of the following: body line or body curvescanning based on multiple techniques, e.g., edge detection, statisticaledge detection, IR camera, microwave sensor, 3D camera, single cameraand multiple cuts when a user is turning around. A complete 3D model ofthe body can also be taken when the user is turning in front of thevideo camera and the user's clothes can be extracted to an alpha channelas described in the technique depicted in FIG. 4, i.e., Colograma, andthe like.

Other learning or authentication methods include, for example, clothdetection, whereby a signature to an item is created and sorted in adatabase. The database can be updated from the internet, by scanningclothes from multiple databases and ecommerce stores or by activelyscanning and updating information by the operator, for example, auniform of security personnel, the type of fabric and the like can beentered.

Also, the combination of EyesMatch with face recognition can allow theoperator to install a camera above a user's height, for example, above adoor facing down about 30-45 degrees; such installation allows the userto move freely below the camera. In this scenario, for example, thedetectable face recognition range is about 1-5 meters in front of thecamera. This is an advantage since known face recognition systems arenot very accurate if the user's head is scanned at an angle of +/−15degrees.

Additional body authentication sensors can include voice, smell, palm,fingerprint, eyes, DNA, skeleton with X-ray sensors or ultrasound,teeth, hair, impression, color, eyes, blood, halo, temperature, skinmarks, ears and the like.

The video, stills or the analyzed data can be recorded with avideo/still recording module 605 directly from the camera, from theEyesMatch transformation module 604, from the processed image or fromdata out of the sensors.

The visual video/image or analyzed data can be displayed in front of theuser screen (full body or smaller), delivered over the cloud ordelivered directly to a control center to be displayed on a screen 606or to a module for further processing and analysis.

Modules 607, 608, 609 and 610 are similar to modules 107, 108, 109 and110, respectively (see, FIG. 1 and associated descriptions thereof).

FIG. 7 depicts an example of a security system application for parallelor in-band camera manipulation.

The capability of the device in FIG. 7 can reside in a stand-alonedevice as depicted in FIG. 7 or as part of the camera DSP capability;the control of the feature activation can be done wirelessly or viawired infrastructure. In addition, some of the capability can besupported in a remote location, e.g., measurement and tracking. Inaddition, the device can be deployed in front of the camera and canproject a manipulated image on a small screen to the camera.

The camera main stream can feed the device or the device can be adaptedfor taping and sniffing the camera main stream. After processing, thedevice can be adapted to send parallel data to the operator.

The device can have multiple image computing capability, EyesMatchgeometric capability, e.g., in the EyesMatch transformation module 703,body measurement and user authentication, e.g., in the security scanningmodule 704, and augmented reality capabilities, e.g., in the augmentedreality module 706, where the image can be manipulated on the mainstream or in parallel to the main stream, e.g., manipulations caninclude a change of a user's body type, color, items held by the user,hair style, complete user disappearance and the like. These capabilitiesare very important for security usage.

In addition, the video stream can feed another device like a robot, ahead mounted display, a user application and the like.

The interface to control the device functionality can be over the wireor wireless infrastructure or to a local or remote location. Themeasurement module can reside in the device or in the cloud.

In one embodiment, the capability, for example, can be done in real-timeor offline. The activation of the device can be done, for example,periodically, in response to a trigger event or manually, as needed ordesired.

The device can support real-time control and triggers for other devices,e.g., a projector for changing the image the user may see (camouflage),or a trigger event for an accurate threat eliminating device that needsan accurate coordination in front of the camera.

Video can be shared real-time with additional stations. There aremultiple ways to share the video. Some exemplary scenarios are discussedin detail below.

Sharing the mirror experience with one or more other users remotely,where the user see himself on a local station and the remote user seethe user in the mirror and share with him the experience. The remoteuser can have any suitable device; the main experience is to see theuser who is standing in front of the mirror and is doing something forreview. The user will be able to see a mirror image of himself based onEyesMatch or any other technology. FIG. 8a depicts this scenario. Theremote user can just see the user broadcast, text the user, speak to theuser and/or be seen in a small window on the mirror screen.

FIG. 8a depicts a sharing mirror experience in a video/voice conference.As shown in FIG. 8a , the mirror experience can be shared live withuser/users in the cloud or via another mirror station. In the firstcase, the user can see himself in a mirror mode although the videostream that goes to the cloud needs to pass an additional mirror flipback to correct the gaze orientation on the other side. So, when theuser (in mirror mode) looks right or left, his eyes/gaze on the otherside will move to the right direction as if he looked at the rightdirection.

FIG. 8b depicts sharing a full body video conference. As opposed to aregular video conference where the camera can be streamed “as is” to theother side, in mirror station, the image that is being sent to the otherside needs to be flipped left-right on the remote location. This can bedone by flipping the mirror captured locally or on the remote side.

In FIG. 8b , when two or more users communicate with each other in thefull body stations, the user/users in one station will see theuser/users from the other direction in the right orientation as well. Asshown in FIG. 8b , the user can see himself standing on the left side ofthe local screen, and the user can see the other user on the right sideof the local screen. In this case the local video is already mirrored,an additional flip/mirror will be needed when streaming the video to theremote location/locations (when broadcasting).

Any combination of mirroring, flipping, rotation and the like of thelocal or streamed video to match this experience of the local mirror andgaze correction on the other remote location are part of the invention.

In one embodiment, both stations are EyesMatched/calibrated to the samegeometrics and dimensions; every one of the stations creates a full bodymirror and sends it to the remote location/locations. The full-body andEyesMatch experience will occur on the video conference as well, and theuser will get the feeling they are looking into each other's eyes.

In one embodiment, where there is more than one user in one location,the EyesMatch or body distortion correction can be done per user withtracking capability per user or for both simultaneously. In order toeliminate the line effect, in this case, augmented reality capabilitycan replace the background as described above. In the augmented realitymode, Colograma or another technique can be used to replace thebackground with a plain background. The connection line can be placedbetween the users to eliminate lines or discontinuity with respect tothe users themselves.

In one embodiment, where there are more than two stations, the screencan be divided to allow multiple users from multiple locations to bedisplayed at the same time, or multiple screens can be placed one nextto the other as depicted, for example, in FIG. 8c . Specifically, FIG.8c depicts multiple mirrors/full body stations with split screen ormultiple screens. As shown in FIG. 8c , the order the users are depictedin the mirror allows for correct gaze communication. When the user islooking to the right, for example, on the remote screen, after the flip,he will be seeing the user looking left toward the remote user.

This is just an example and any number of users near the screen or inremote location should be organize to allow for right gaze approach. Inone embodiment, further gaze and eye contact improvement can be appliedto improve the experience. In one embodiment, the screen can include 3Dcapability, and 3D capability per user in the screen.

In one embodiment, in addition to the EyesMatch or any computerizedimage processing at one side, the remote location can have additionalcomputerized capability to manipulate on the far end video based on thelocal user tracking, e.g., if the user gets closer to the screen, thefar end video or image will be processed to look a bit bigger andprojected to provide a better feeling of a live session. Another exampleis when a user gets closer on one end, he will see a wider FOV of thefar end.

User Interface

The present invention includes virtual mirror control and virtual systemfeatures to ease the usability of the virtual mirror. The virtual mirrorcontrol and virtual system can include various modes of operationincluding, for example, on, off, restart, mirror, display, idle, and thelike. The virtual mirror control and virtual system can includeauthentication and registration of users, which can be automatic, manualor a combination of automatic and manual. The virtual mirror control andvirtual system can include an optimal flow of operation to ease andenhance the user experience.

The virtual mirror control and virtual system can include one or moreinterfaces to control the mirror, e.g., automatic gesture, voice, eyepointing, mobile application, remote expert, local assistance, and thelike. The virtual mirror control and virtual system can include acomprehensive combination of interfaces, including the following:playing a recorded video (automatic/manual); advance effects (item colormanipulation, augmented reality features, gaming features, backgroundchange, 3D effect, lighting effect and the like); screen mode(orientation/full size or split screen); sharing techniques and methodsto share the experience with friends/experts in real-time and on the go;mobile user application for controlling display and sharing, forexample, sharing of personal videos and images taken from the virtualmirror or the user mobile device; retail or enterprise remote forgeneral management (provide retailer information about the user inreal-time with advanced control and analytics capability); and a moduleto teach the user how to operate the mirror (animation, video, voice,visual hints and the like).

FIGS. 14, 15 and 16 depict examples of a virtual mirror user interface(UI) flow that can be sequenced together to simulate a mirrorexperience. Outside operation hours, the mirror screen and/or thecomputational device that feed the screen can be turned off or remain onwith a screen saver to eliminate rapid aging of the display electronics.The mirror station can be enabled so as to automatically power up. Themirror station can be controlled via remote control locally or throughthe cloud to reboot and to enter into normal operation.

When the system is turned on, before a user stands in front of themirror, the system operates in an idle mode, during which the controllermay or may not provide transformation to mimic a mirror reflection. Oncethe user enters a specified zone in front of the field of view of thecamera, the processor is capable of tracking the user in the recordedimage. Based on the tracking, the video engine of the processor cancalculate a transformation of the image to mimic mirror behavior.

In the idle mode, normally operable when there is no user in front ofthe mirror, monitor may display a screen saver, a static image, videoclips, and the likes. All of these examples may be referred to as idleimage, wherein the term “image” includes video. During idle mode themonitor may also display the background image recorded at thatparticular instance by the camera. In order to properly display thebackground image, so that it appears like a mirror reflection, the videoengine of the controller can take a default setting (e.g., 2 meter fromthe monitor) and apply the 2 meter transformation on the camera streamto create a mirror effect on the background environment as depicted, forexample, in FIG. 14.

In various embodiments, the system is capable of autonomously recognizethe presence of a user in front of the camera and automatically startmirror mimicking procedure, including user identification, accountmatching, etc. In one embodiment, the presence of a user is done bycontinuously analyzing the images captured by the camera to detectchanges in the image and identify a user. In one example, a trigger zonecan be designated by placing a mat or using a specific indicia on thecarpet in front and within the field of view of the camera. The carpetcan be designed with a specific pattern and color that improves thetracking and detection of the user. In addition, the colors of thecarpet can improve the quality of the video by, e.g., improving thespecific contrast. For example, lighter colors can be used to improvelighting conditions and improve the quality of the resulting video. Inone embodiment, the pattern on the carpet can be alternating at around5-10 cm with lighter and darker colors so that when a user stands withdark or light color shoes, the user shoes will be easily detected andtracked, and the maximum distance error will not be higher than half thecolor separation, i.e., 5-10/2. In one embodiment, the carpet caninclude instructions for the user informing the user how to operate themirror. Alternatively, in one embodiment, an operation device can beprovided where a user can step with their feet on a receptive pad tooperate the mirror with their feet rather than their hands or voices. Inone embodiment, the mirror can prompt the user to stand in the rightlocation, e.g., when a user gets closer, the mirror can switch back toidle mode or the mirror can cut the user legs in the image, and thelike.

The system can be configured so that, when the user steps into thetracking or registration zone in front of the mirror, the video engineof the controller reacts and starts tracking the object. Based on theobject location, the video engine can adjust the video transformation tomimic mirror behavior. Additional input to the tracking can include theuser's height, spatial location, pose and the like, as described in theEyesMatch process. At this point, the user is still not registered tothe system, so an additional process of user identification can beperformed. For example, the authentication can include face recognition,a special card that a user can present to the system for scanning, auser mobile device (audio, wireless, QR (Quick Response) code), or otherbiometric registration feature.

In one embodiment, since facial recognition is not entirely reliable100% of the time, additional recognition features can be displayed so asto pop up on the screen for an appropriate amount of time, such as a fewseconds. In the current example, a QR bracket 1505 (FIG. 15) is openedand the user can show the mirror a QR badge or QR that was sent to hisunique application, e.g., on a smartphone, and effectively communicateto the system that the present facial image is associated with theauthenticated user. This technique can be used to allow semi-automaticuser merging in the event a particular user has more than one facialaccount. In addition, once facial information is merged, additionalfacial measurements and/or improvements can be performed to improve therecognition of the user over time. That is, when the user isauthenticated using the QR code, all of the facial informationassociated with this specific code, from the current and all pastsessions, are used to update the face recognition information for thisuser.

In one specific example, illustrated in FIG. 15, the image presented onthe monitor includes a frame 1505 superimposed on the video image. Thus,the user looking at the monitor sees his artificial “reflection” on themonitor and the superimposed frame or bracket 1505. The user may thenpresent a code, such that the code fits within the frame on the monitor.In one example the code is a printed barcode on a card. According toanother example, the user may download an app onto the user's mobiledevice, e.g., a smartphone. The app includes a barcode, such as a QRcode, specifically corresponding to the user or the user's smartphone.When the user sees the frame, the user opens the app and presents thesmartphone such that the smartphone fits within the frame. Thecontroller then identifies the code within the frame and therebyidentifies the user.

When the user is identified, the user's account can be opened and thelast recordings can be displayed, e.g., in one embodiment, a thumbnailconfiguration 1510 can be displayed, such as that depicted in FIG. 15.Alternatively, any other image control bar can be displayed. If the useris not identified, a user registration process can commence, then, aftera few seconds, a new account can be opened and the mirror can beconfigured to start recording automatically. For example, if the user isnot identified, a code, such as a QR can be displayed on the monitor,such that the user can scan it with a mobile device to download the app.When the app is downloaded and the user completes the registrationprocess, the app on the user's device would include a code that can bepresented to the frame in future visits.

In one embodiment, the system can record and store a video clip of Nseconds, e.g., 10-13 seconds is enough time to allow the user to get agood impression on the way a particular item looks and can include,e.g., the user turning around and the like. So, while the user maycontinue to inspect the current item on the mirror, and the controllermay continue to provide transformation for mirror mimicking, the systemrecords only a sub-section of the entire session. This sub-session maybe stored in the database, e.g., on the cloud, and be available duringfuture sessions of the same user, be available for download onto theuser's device, and/or may be available for sending by the user to otherusers or devices. For example, the user may be able to upload thesession to social media or other applications.

In one embodiment, although the mirror can be configured to startrecording automatically when it recognizes the user, if the user doesnot want such auto-recognition, the mirror can be configured to locallysave the video, the user can trigger one of the previous thumbnails, themirror can be configured to switch to a display mode and the mirror canbe configured to play the video. That is, the user can be prompted toexecute a process by which a particular scene is cut by the user and notsaved in the system.

In one embodiment, one or more thumbnails 1510 can be added on themirror, which can be useful when the screen is in a narrowconfiguration, e.g., 21:9, or any other wide screen configuration. Inone embodiment, the thumbnail 1510 can be displayed so as to pop up in aseparate bar next to the mirror stream or under the mimicked videostream. This feature can be useful when the screen is wider thanrequired or proportioned for the mimicked mirror stream. In oneembodiment, six (6) thumbnails 1510 are presented to the user, thoughany suitable number of thumbnails can be presented. The size of thethumbnails can be configurable to support a reasonable proportion forthe display. Each thumbnail 1510 includes a hyperlink to a prior sessionof the user, which can be accessed by activating the hyperlink. Thehyperlink may be activated by various methods, such as, e.g., handmotion, mouse, remote control, etc. In one example, the controlleridentifies the user's hand in the image. As the user moves the hand upand down, the controller follows the motion and highlights acorresponding thumbnail. When the user closes the user's hand to form afist, the controller activates the hyperlink corresponding to thethumbnail that was highlighted at that particular moment.

In one embodiment, as can be seen, for example, in FIG. 15, a recordingindicator 1520 can be displayed as well. The recording indicator caninclude display of a phrase like “REC”, a red colored circle and thelike.

Also, text or user's name can be displayed as shown in 1515 in FIG. 15.In one embodiment, an additional password can be associated with theuser and displayed as text 1515, and the user can take thispassword-based authentication and apply it to the registration process.In one embodiment, an employee or the user himself can enter a phone, anemail, NFC signaling or any other identification information and obtaina link to the database on the cloud on the spot or later.

In one embodiment, additional control capability can be added as well,e.g., start recording, erase video, add video effects like lightening,color changes, background selection and the like. In one embodiment, allthe videos can be uploaded to the cloud and can be erased from the localstation after some preprogrammed period of time. In one embodiment, thevideos can be processed after recording with additional effects thatimprove the video or change the video to reflect augmented realityeffects or for BMI/tailor/body line measurements analysis. In oneembodiment, the video recording can include the audio recording of theenvironment for further analysis. In one embodiment, the video recordingcan include WLAN recording of the environment to record the MAC of theusers and later on with further correlation, can be adapted to associatethe MAC of the mobile device to the user.

The present invention includes a system for facilitating a user'scontrol of the mirror. In one embodiment, if the user is still in thepredefined zone and was recognized by the mirror once, as long as he isstill standing there and no one else is being recognized, the user willbe able to control the mirror, e.g., play, start and stop images/videos,delete images/videos, add augmented reality features and the like. Theuser can also control the mirror through gesture control or from hisdevice, e.g., smartphone, through a dedicated application or additionalcontrol features as part of the retail application. In one embodiment,the gesture control can enable some basic functionality and the userapplication can be allowed to enable much more functionality relative tothe gesture control. In one embodiment, the assistant in the store or anassistant available virtually/remotely in the cloud can assist the userin operating the mirror. In one embodiment, the user can set his ownpreferences for the UI from his application or from the web. All userdedicated settings can be added to the user's account, e.g., the usercan change the time length of a session recording, the number ofthumbnails, and the like.

In a default mode, the user can see his last n videos. The videos may ormay not have been taken in the specific mirror location, but may beaccessed at a central storage location, e.g., from the cloud. The usercan set the appearance of the thumbnails of the local try-on videos.From his application, the user can see all the thumbnails and be able totouch them or any other election method to activate play of the videoassociated with the specific thumbnail.

In one embodiment, the thumbnails 1510 are configured so as to pop up ontop of the mimicked mirror stream as can be seen, for example, in FIG.15. The thumbnails can be sent to the background when the real-timemirror-mimicking video is playing or in split mode when there is nogesture control identification as shown, for example, in FIG. 16. Theuser can also set the screen to a split mode as shown, for example, inFIG. 16. Specifically, FIG. 16 depicts a side-by-side setting where theuser can still see himself in real-time on one side of the screen andcan play previously recorded videos on the other half.

In one embodiment, the remote control of the user can obtain theidentified user information automatically so the store assistant cansend the user link to his account for registration or to help the userwith additional capability like changing clothes colors or changingclothes in general.

In one embodiment, the user can use his mobile device to initiate avideo call in the mirror with a remote expert or a friend that canadvise him in real time. The call can be from a dedicated application orany other third-party application like Skype. The mirror video streamcan be fed to the local Skype and the remote user can get the mirrorstream in real time. For example, the real-time or storedmirror-mimicking video can be send to the user's device using, e.g.,WiFi connection. The app then enables coupling the downloaded orstreamed video to the communication app, such as Skype. On the otherhand, the app may enable to user to send a link via, e.g., email or SMS,to enable a third party cloud access to the streaming mirror-mimickingvideo in real time.

FIG. 17 depicts one embodiment of a suggested flow of usage.Specifically, FIG. 17 depicts an embodiment of user flow of usage of theadditional embodiments depicted in FIGS. 14, 15 and 16 and theirassociated descriptions. In step 1701 the mirror is in idle position,during which time an idle image is presented on the monitor. The idleimage may be, e.g., a screen saver, commercials, slide show, or just animage of the field of view in front of the camera and transposed by thecontroller to mimic a mirror reflection of the field of view.

In step 1702, as a user approaches the mirror and the user's presence issensed, e.g., by motion sensor or by detecting a change in the imageseen by the camera, the system initiates operation in mirror mode. Thatis, the controller perform transformation operation on the image of theuser, such that the image presented on the monitor mimics the user'sreflection in a mirror. At step 1703, the system initiatesauthentication procedure to identify and authenticate the user. Forexample, in one embodiment the system uses face recognition to identifyand authenticate the user, while in another embodiment the user may beauthenticated using a device such as a smartphone with capabilities suchas, WiFi, Bluetooth, NFC, etc. For example, in one embodiment facerecognition is used to identify the user, but a second step, e.g., useof QR code is used to authenticate the user. If the user has not beenidentified, e.g., it is a new user, the user may be prompted to open anaccount. For example, a QR code may be presented on the monitor,enabling the user to scan the QR code using a mobile device, to therebydownload an app to the user's device. Otherwise, or if the user declinesto open an account, the user may proceed as a guest, albeit somefeatures, such as remote access to stored images, may not be availableby guest access.

In step 1704 the user is given user control over the display. In oneembodiment, the specific control preferences are saved for eachparticular user and are activated once the user has been recognized.Otherwise, general user interface is enabled, e.g., a hand gestureactivated interface. In one embodiment, when a user has been recognized,or has tried on several outfits in the same session, prior trials arepresented as thumbnails on the side of the main display. When the userraises the hand corresponding to the side of the thumbnails, dependingon the height of the hand, a corresponding thumbnail is highlighted forselection. As the user raises or lowers the hand, another thumbnailcorresponding to the height of the hand is highlighted for selection.Then, when the user closes his fist, the highlighted thumbnail isselected and the corresponding image or video is displayed on the mainscreen.

As shown in 1705, the images and video of each session may be stored onthe cloud, and may be accessible remotely using, e.g., smartphones,tablets, etc. Thus, a user may converse with friends remotely and gettheir opinion on the outfit the user is trying. For example, the usermay send the friend a link to the recorded trial or to a live streamingvideo from the system. Thus, the user may share the shopping experiencewith people located remotely.

Moreover, as also shown in 1705, since the system is able to identifythe user and also calculate parameters of the user, e.g., weight,height, etc., the system may be able to access a database of availableitems that would be recommended to the user based on these parameters.More specifically, if the user has recorded two trials of two differentshirt within the same session, the system can decipher that the user isinterested in purchasing a shirt and make either alternativerecommendations, i.e., different shirts, or complimentaryrecommendations, e.g., specific pants that go well with the tried onshirts. Also, since the system can identify the shirt and the brand ofthe shirt, it may be able to offer specific incentive from thatmanufacturer, as exemplified in 1706.

Also, in 1706 the user may be offered the ability to change colorswithout having to actually change the garment. As explained above, bythe use of multiple masks or layers, the system is able to change thecolor of the article, while maintaining the realistic visual effect ofthe fabric as it is worn by the user. Since the system may maintain alayer of shading, a layer of texture, a layer of reflection, etc., thecolor layer may be changed while maintaining all the other layers, suchthat the rendered image retains all of the characteristics of the priorimage, with only the color changed.

In order to avoid complication of the flow of the experience in front ofthe mirror, a complimentary application can enable additional featuresand settings, e.g., an advanced user that is comfortable with gesturecontrol can enable more gestures that will help to operate advancedfeatures without use of a tactile interface application. For users thatare not comfortable with hand gestures, such users can be provided withbasic automated features and the mobile application can be used tooperate the other advanced functionality.

In one embodiment, the mirror station can be configured to identify theuser's behavior and adapt the gesture control to the way the user istrying to operate it, e.g., some users will point, some users will grab,and some users will push to operate a thumbnail. The system can beadapted to learn and update a profile control per user.

In one embodiment, the mirror station can support instruction to theuser, so that when the system identifies a user that is not controllingthe mirror in the correct manner, a short explanation can pop up andguide the user through the process.

In one embodiment, the mirror can be adapted to demonstratepredetermined characteristics based on information regarding the user'sfamiliarity with the system, e.g., based on whether the user is a firsttime user, based on the number of videos in the system for the user,based on the last time the user required the system to enableinstructions, can be adapted to provide relatively simpler functionalityvoice assistance, can be adapted to provide remote expert assistance andthe like.

In one embodiment, the concept of a pre-defined number of thumbnails, asshown in FIGS. 15 and 16, can be replaced with a slide menu ofthumbnails.

In the normal operation mode as depicted, for example, in FIGS. 15 and16, the user can move his hands up and down to select thumbnails, oncethe user's hand stops, the closest thumbnail can be elected and can bedesignated such that the user will get feedback for the selection. Forexample, blue shade can be provided around the selected thumbnail tosignify its election. When the user grabs, pushes, or points hishands/fingers at the thumbnail, the video can start playing based on thedisplay mode setting, and the user can also stop the video and performother operations and the like. Once the user, for example, stops theplayed video, a hold bar can be displayed on the thumbnail to designatethe status of the played video and to acknowledge back to the user thathis command was accepted by the mirror station.

In one embodiment, to simplify the operation further, the gesturecontrol by hand can be configured to detect a jump between a thumbnaileven if the user is moving his hands left and right instead of up anddown or any other direction. The system can be configured to adaptitself to the user's span and velocity of hand movements, so that theuser can see the election so he can adapt his speed and span of movementas well.

The embodiments may be implemented in a non-transitory computer-readablestorage medium for operating a monitor, a camera, and a processor, so asto display a mirror-mimicking image on the monitor, comprising: on adevice having the processor and a memory storing a program for executionby the processor, the program including instructions for: sensing for auser; initiating a mirror-mimicking mode for displaying themirror-mimicking image on the monitor; initiating an authenticationprocess; and prompting the user to control the monitor.

Augmented Merchandizing

The mirror system can be used to provide augmented merchandizing byenabling better service to users, promoting merchandize, and providingfeedback to merchandizers. The mirror can be configured to complementand augment the business process. Since the mirror station according tothe present invention is a new device in the marketplace, uses of themirror to create a revenue stream is unique as well. The next fewembodiments include details providing examples as to how an activemirror can be used to augment the business and to create new a revenuestreams.

For example, in one embodiment, the logo of a specific brand or partneror any other logo can be added to the recorded video, so the user willget, e.g., video with the logo of the brand, and the user can share theresulting video with his friends. Consequently, a single session infront of the system may be used to promote a certain item and/or brand.For example, if the user uploads the session to a social network site,the session may be seen by many people, all of who would be exposed tothe superimposed logo.

In one embodiment, the mirror can be configured to provide convincingaugmented reality capability by an item that was previously tried on byanother user with a similar body type. This matching technique yields amuch more convincing experience and provides better recommendations tothe user.

The mirror can insert commercials, coupons and logos into the mirror inidle mode, within a specified portion of the mirror, or via split screenmode. The mirror can also introduce commercials, coupons and logos intorecorded videos so a user can view the video with thecommercial/coupon/logo. These displays can be sold as advertising. Theuser can share the commercial/coupon/logo by e.g., uploading a sessionto social network sites. The user can be offered an incentive inexchange for sharing the commercial/coupon/logo.

The user's try-on session of clothing can be analyzed for measurementsand/or characteristics such as BMI, sex, origin, age, body measurements,facial expression, voice expression, recommended sizes and the like.This data can be incorporated into ecommerce applications. This data isof high value and can be shared with brands, the user, a third partybased on an agreement with the user and the brands, etc. Utilizing thepresent invention, a revenue stream can be generated from the analyzeddata regarding the user, for example, by presenting the user withfocused and accurate advertising according to recorded sessions.

The mirror can be configured as a service that allows a user to engageexperts, a friend or an advisor when the user contemplates a particularpurchase. Additional revenue can be generated using the presentinvention by promoting up sale opportunities related to the advice ofthe remote expert. The experts can be rated by the user, and user canpick his own favorite expert or select an expert designated by thebrand.

The user can be presented with a display including a suggestion and/orincentive for a particular item tried on by the user, for example, withdifferent colors. This method presents a further opportunity to promoteup-selling. Augmented reality can be used to promote up-sales.Specifically, based on information obtained by the mirror and other usertries, a more compelling augmented reality display can be generatedbased on a combination of information about the user and similar usersthat have a similar body type.

In one embodiment, the social network of the user can be integrated in amanner to help the user and/or comment on the user's contemplation ofthe item. The system can be configured to share the user's videos. Thesocial networking features can be used to enlarge the database ofaddressable customers.

In one embodiment, the user can upload their own photos, the videoengines can be configured to process the images and provide similaranalysis of BMI, face, origin and the like. The system can providerecommended sizes for ecommerce applications. The system can show theitem that was bought in different colors. In this manner, the mirror canbe used to obtain marketing information for an appropriate database evenwhen the user isn't physically standing in front of the mirror itself.

In one embodiment, the user can see his predicted size in theapplication and update the measurements, which can be used to improvethe size prediction model and include adjustments for particular brands.

Physical Design

The present invention includes the mechanical design and appearancedesign of the equipment. The screen can be mounted so as to stand on thewall either vertically or horizontally or can be switchable betweenvertical and horizontal (by a larger or similar mechanical solution, thescreen can be tilted, rotated and the like). The screen can be mountedon a dedicated stand, mounted on a wall or mounted behind the wall. Whenthe screen is inside the wall, thermal ventilation duct should beprovided to support the screen and the computing device.

In one embodiment, the screen can have a proportion of a mirror, e.g.,21:9.

In one embodiment, the screen can have a regular proportion, i.e., 16:9,and the setting of the display can be split screen or dark bars on thesides to yield the mirror proportion, i.e., 21:9.

In one embodiment, the screen can come in a matte finish to eliminatereflections that can be seen on dark colors. In other words, when theuser wears black, and the screen is not matte-finished, the user can seehis own true mirror reflection; this effect is not wanted and candiminish, distort or completely ruin the virtual mirror effect.

The frame can be presented in a manner as depicted in FIGS. 14, 15 and16. A minimal frame can be used or the frame can be hidden behind awall.

In one embodiment, the cameras or the controlled cameras can be locatedin the frame. The camera can be mounted above the screen frame, can behidden and can be mounted on the wall. The camera can be provided with ashade that can be filtered out from the image. Since the camera isfacing down, a cover that will hide most of the camera body can blockthe view from a distance.

The screen can be LED, LCD, plasma, glass, projector and the like.

A background can be used that enables better white balancing thatresults in higher quality videos.

Lighting can include a combination of white, yellow or any combinationin a spot or a projector combination and can be configured to improvethe video quality and the colors.

A dedicated carpet can be used to allow background changes, to define auser zone, to improve user detection and tracking in front of themirror, to instruct the user where to stand and to instruct the user onhow to operate the mirror.

CONCLUSION

Each of the above identified modules or programs corresponds to a set ofinstructions for performing a function described above. These modulesand programs (i.e., sets of instructions) need not be implemented asseparate software programs, procedures or modules, and thus varioussubsets of these modules may be combined or otherwise re-arranged invarious embodiments. In some embodiments, memory may store a subset ofthe modules and data structures identified above. Furthermore, memorymay store additional modules and data structures not described above.

The illustrated aspects of the disclosure may also be practiced indistributed computing environments where certain tasks are performed byremote processing devices that are linked through a communicationsnetwork. In a distributed computing environment, program modules can belocated in both local and remote memory storage devices.

Moreover, it is to be appreciated that various components describedherein can include electrical circuit(s) that can include components andcircuitry elements of suitable value in order to implement theembodiments of the subject innovation(s). Furthermore, it can beappreciated that many of the various components can be implemented onone or more integrated circuit (IC) chips. For example, in oneembodiment, a set of components can be implemented in a single IC chip.In other embodiments, one or more of respective components arefabricated or implemented on separate IC chips.

What has been described above includes examples of the embodiments ofthe present invention. It is, of course, not possible to describe everyconceivable combination of components or methodologies for purposes ofdescribing the claimed subject matter, but it is to be appreciated thatmany further combinations and permutations of the subject innovation arepossible. Accordingly, the claimed subject matter is intended to embraceall such alterations, modifications, and variations that fall within thespirit and scope of the appended claims. Moreover, the above descriptionof illustrated embodiments of the subject disclosure, including what isdescribed in the Abstract, is not intended to be exhaustive or to limitthe disclosed embodiments to the precise forms disclosed. While specificembodiments and examples are described herein for illustrative purposes,various modifications are possible that are considered within the scopeof such embodiments and examples, as those skilled in the relevant artcan recognize.

In particular and in regard to the various functions performed by theabove described components, devices, circuits, systems and the like, theterms used to describe such components are intended to correspond,unless otherwise indicated, to any component which performs thespecified function of the described component (e.g., a functionalequivalent), even though not structurally equivalent to the disclosedstructure, which performs the function in the herein illustratedexemplary aspects of the claimed subject matter. In this regard, it willalso be recognized that the innovation includes a system as well as acomputer-readable storage medium having computer-executable instructionsfor performing the acts and/or events of the various methods of theclaimed subject matter.

The aforementioned systems/circuits/modules have been described withrespect to interaction between several components/blocks. It can beappreciated that such systems/circuits and components/blocks can includethose components or specified sub-components, some of the specifiedcomponents or sub-components, and/or additional components, andaccording to various permutations and combinations of the foregoing.Sub-components can also be implemented as components communicativelycoupled to other components rather than included within parentcomponents (hierarchical). Additionally, it should be noted that one ormore components may be combined into a single component providingaggregate functionality or divided into several separate sub-components,and any one or more middle layers, such as a management layer, may beprovided to communicatively couple to such sub-components in order toprovide integrated functionality. Any components described herein mayalso interact with one or more other components not specificallydescribed herein but known by those of skill in the art.

In addition, while a particular feature of the subject innovation mayhave been disclosed with respect to only one of several implementations,such feature may be combined with one or more other features of theother implementations as may be desired and advantageous for any givenor particular application. Furthermore, to the extent that the terms“includes,” “including,” “has,” “contains,” variants thereof, and othersimilar words are used in either the detailed description or the claims,these terms are intended to be inclusive in a manner similar to the term“comprising” as an open transition word without precluding anyadditional or other elements.

As used in this application, the terms “component,” “module,” “system,”or the like are generally intended to refer to a computer-relatedentity, either hardware (e.g., a circuit), a combination of hardware andsoftware, software, or an entity related to an operational machine withone or more specific functionalities. For example, a component may be,but is not limited to being, a process running on a processor (e.g.,digital signal processor), a processor, an object, an executable, athread of execution, a program, and/or a computer. By way ofillustration, both an application running on a controller and thecontroller can be a component. One or more components may reside withina process and/or thread of execution and a component may be localized onone computer and/or distributed between two or more computers. Further,a “device” can come in the form of specially designed hardware;generalized hardware made specialized by the execution of softwarethereon that enables the hardware to perform specific function; softwarestored on a computer-readable medium; or a combination thereof.

Computing devices typically include a variety of media, which caninclude computer-readable storage media and/or communications media.Computer-readable storage media generally can be any available storagemedia that can be accessed by the computer, is typically of anon-transitory nature, and can include both volatile and nonvolatilemedia, removable and non-removable media. By way of example, and notlimitation, computer-readable storage media can be implemented inconnection with any method or technology for storage of information suchas computer-readable instructions, program modules, structured data, orunstructured data. Computer-readable storage media can include, but arenot limited to, RAM, ROM, EEPROM, flash memory or other memorytechnology, CD-ROM, digital versatile disk (DVD) or other optical diskstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, or other tangible and/or non-transitorymedia which can be used to store desired information. Computer-readablestorage media can be accessed by one or more local or remote computingdevices, e.g., via access requests, queries or other data retrievalprotocols, for a variety of operations with respect to the informationstored by the medium.

Communications media typically embody computer-readable instructions,data structures, program modules or other structured or unstructureddata in a data signal that can be transitory such as a modulated datasignal, e.g., a carrier wave or other transport mechanism, and includesany information delivery or transport media. The term “modulated datasignal” or signals refers to a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin one or more signals. By way of example, and not limitation,communication media include wired media, such as a wired network ordirect-wired connection, and wireless media such as acoustic, RF,infrared and other wireless media.

In view of the exemplary systems described above, methodologies that maybe implemented in accordance with the described subject matter will bebetter appreciated with reference to the flowcharts of the variousfigures. For simplicity of explanation, the methodologies are depictedand described as a series of acts. However, acts in accordance with thisdisclosure can occur in various orders and/or concurrently, and withother acts not presented and described herein. Furthermore, not allillustrated acts may be required to implement the methodologies inaccordance with the disclosed subject matter. In addition, those skilledin the art will understand and appreciate that the methodologies couldalternatively be represented as a series of interrelated states via astate diagram or events. Additionally, it should be appreciated that themethodologies disclosed in this specification are capable of beingstored on an article of manufacture to facilitate transporting andtransferring such methodologies to computing devices. The term articleof manufacture, as used herein, is intended to encompass a computerprogram accessible from any computer-readable device or storage media,e.g., stored on a server coupled to the cloud.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tobe limiting to the precise forms disclosed. Many modifications andvariations are possible in view of the above teachings. The embodimentswere chosen and described in order to best explain the principles of theaspects and its practical applications, to thereby enable others skilledin the art to best utilize the aspects and various embodiments withvarious modifications as are suited to the particular use contemplated.

We claim:
 1. A computer implemented method for operating a system havinga monitor, a camera, a memory, and a processor, so as to display amirror-mimicking image on the monitor to enable a user to try a clothingitem, comprising a program for execution by the processor, the programincluding instructions for: at each clothing item try session performingthe steps: obtaining digital images of a live video feed of length of Nseconds from the camera; flipping the images about a vertical axis so asto reverse right and left sides of the image; applying a transformationmapping to the images to modify the images such that they appear tomimic a reflection of a mirror, to thereby obtain transformed images;saving the transformed images in the memory; displaying the transformedimages on the monitor to thereby replay the video feed of N secondsusing the transformed images; wherein the length of N seconds isconfigured to be enough time to allow the user to get a good impressionon the way the clothing item looks.
 2. The method of claim 1, furthercomprising enabling the user to stop the replay of the video.
 3. Themethod of claim 1, further comprising storing on the cloud thetransformed images of the video of each session.
 4. The method of claim3, further comprising enabling remotely accessing the transformed imagesof the video stored in the cloud.
 5. The method of claim 1, furthercomprising enabling the user to send a link to the video stored in thecloud.
 6. The method of claim 1, further comprising enabling the user tostream the video using the transformed images from the cloud.
 7. Themethod of claim 1, wherein the length of N seconds is configured to beenough time to allow the user to turn around.
 8. The method of claim 1,further comprising instructions for at least one from the groupconsisting of: changing a background of the image or video; changing theappearance of an object of clothing or apparel worn by the user with aselectable virtual object; and adding an accessory to the appearance ofthe user.
 9. The method of claim 1, wherein the program furthercomprising instructions for when the user steps into atracking/registration zone in front of the monitor, start tracking theuser.
 10. The method of claim 1, wherein the program further comprisinginstructions for generating a model mask to enable changing a color ofthe clothing item.
 11. The method of claim 10, wherein the model maskcomprises a plurality of layers, having at least color layer and texturelayer.
 12. The method of claim 11, wherein the model mask furthercomprises a grey scale mask of the clothing item.
 13. The method ofclaim 1, wherein the program further comprising instructions forgenerating a split screen on the monitor where the user can see himselfin real-time on one side of the screen and can play previously recordedvideos on the other half.
 14. The method of claim 1, wherein thetransformation is configured to match the camera point of view with atheoretical mirror point of view.
 15. The method of claim 1, wherein theobtaining digital images commences upon sensing a trigger event.
 16. Themethod of claim 1, further comprising restarting the play of the videofeed according to a user input.
 17. The method of claim 1, wherein theprogram further includes instructions for erasing the stored video fromthe local storage after a preprogrammed period of time.
 18. A system forenabling a user to try a clothing item, comprising: a monitor; a cameragenerating digital images; a processor comprising a video engine; atrigger event module; wherein the processor is configured to, uponreceiving a trigger event, obtaining the digital images of a video oflength of N seconds from the camera, the length of N seconds isconfigured to be enough time to allow the user to get a good impressionon the way the clothing item looks; and wherein the video engine isconfigured to apply a transformation to the digital images of the videofeed so as to replay the video feed of N seconds using the transformedimages; a memory for storing the transformed images.
 19. The system ofclaim 18, wherein the trigger event module comprises a trigger zonedesignated in front and within the field of view of the camera.
 20. Thesystem of claim 18, wherein the system is configured to record only asub-section of an entire session of an item try by the user.
 21. Thesystem of claim 20, wherein the sub-session is stored on the cloud, andmade available for download onto the user's device.
 22. The system ofclaim 21, wherein the sub-session is made available for sending by theuser to other users or devices.